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MULTIMEDIA DATA STRUCTURE REFLECTING CHANGE OF USER RELEVANCE 
FEEDBACK AND METHOD OF DETERMINING WEIGHTS OF IMAGE FEATURES OF 
MULTIMEDIA DATA 

5 BACKGROUND OF THE INVENTION 

Field of the Invention 

The present invention relates to a method of determining 
■weights of image features by considering both the recent user 

0 relevance feedback information and the whole feedback 

information in a system which determines the weights of the 
image features using the user relevance feedback, and a data 
structure therefor. 

In particular, the present invention relates to a 

5 multimedia data structure and a method of determining weights 

of image features using the multimedia data structure which can 
reflect both the recent user relevance feedback information 
determined for a predetermined time ' period or by the 
predetermined number of frequency and the information on the 

0 whole user relevance feedback information till now when 

determining the weights of the image features in a multimedia 
search system which changes the weights of the image features 
based on the user relevance feedback if the user gives the 
feedback regarding a similarity with respect to a certain image 

5 and then uses the changed weights of the image features for the 
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future search. 

Description of the Related Art 

Conventionally, in measuring the similarity between images 
5 for an effective image search, a method of determining the 

similarity by giving a weight for each feature, and a method of 
determining the similarity by giving a weight for each element 
in a feature are used. 

Specifically, if the user gives feedback information on a 
10 similar image ' or an image not similar as he/she views images 

presented in response to a user's query, the system 
automatically calculates weights of respective features such as 
color histogram, texture histogram, dominant color, etc., using 
the feedback' information, and uses the calculated weights for 
15 the image search. 

Such learned weights may be stored in table, and used 
again for subsequent searches for other images . 

Also, in case of searching an image having no weight given 
thereto, a previously learned weight of another similar type 
2 0 image may be used. For this, similar image group information is 

managed in a table. 

Specifically, even when the current reference image has no 
weight given thereto, a search is performed with reference to a 
specified weight if any other image in the group, to which the 
2 5 reference image belongs, has such a weight. 

2 



In the image search reflecting the user relevance feedback 
\ as described above, the method .-using the learned weight through^ 
the user relevance feedback has an advantage that the weight is 
the value which reflects the whole feedback till now with 
5 respect to the image, but has a disadvantage that it cannot 

reflect the change of the recent feedback pattern when the 
recent user feedback pattern is changed. 

For instance, if the corresponding image moves to another 
database, a feedback different from the feedback pattern that 
10 the user has reflected till now will be performed. 

In this case, if lots of feedback are reflected as a 
predetermined time elapses after the database movement, the 
feedback pattern produced after the database movement will 
affect as more important information than the previously 
15 reflected feedback pattern. 

However, the above -described weight learning method does 
not provide the result learned using only the feedback within a 
specified period, and thus it can hardly reflect the user 
feedback pattern recently changed in case that it has only the 
2 0 weight learning information according to the whole feedback as 

described above. 

Meanwhile, in case of managing the similar image group 
information in list, all the feedback images cannot be managed 
in list since a large amount of storage space is required, and 
2 5 thus a limited list using only the recent feedback information 
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can be managed. 

In this case, the recent user feedback pattern can_.be' 
reflected, but the whole feedback pattern cannot be reflected. 

For instance, if the weight of the image feature is 
5 determined and learned only by the limited feedback for the 

latest period in the event that the recent user feedback is not 
in consistency, the characteristics of the corresponding image 
cannot be reflected, and the reliability of the image feature 
information greatly deteriorates due to the irregular feature 
10 weight learning. 

SUMMARY OF THE INVENTION 

Therefore, an object of the invention is to provide a 
15 multimedia data structure and a method of determining weights 

of image features which can determine the weights of the image 
features by considering both the recent user feedback pattern 
corresponding to a predetermined time period or the frequency 
number and the whole feedback pattern when the weight of the 
20 image feature is determined using the user relevance feedback. 

Another object of the invention is to provide a 
multimedia data structure and a method of determining weights 
of image features which can determine the weights of the image 
features by considering both the recent user feedback 
25 information corresponding to a predetermined time period or its 
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frequency and the whole feedback information and in proportion 
' to reliability information representing how' reliable the recent 
user feedback information is, reliability information 
representing how reliable the whole feedback information is, 
5 and reliabilities of the two information, when the weights of 

the image features are determined using the user relevance 
feedback. 

In accordance with the present invention, these objects 
are accomplished by providing a multimedia data structure 
10 reflecting change of a user relevance feedback in a system for 

determining weights of features used for an image search using 
the user relevance feedback, the multimedia data structure 
comprising : 

(a) information describing the features of a certain 

15 image; 

(b) recent user feedback information when weights of the 
features of the certain image are learned and determined by the 
user relevance feedback; and 

(c) whole feedback information when weights of the 
2 0 features of the certain image are learned and determined by the 

user relevance feedback; 

(d) wherein the weights of the image features are 
determined considering both the recent user feedback 
information and the whole feedback information, and then used 

25 for search. 
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In accordance with the present invention, the recent 
feedback information . .'and the whole' feedback information/- are 
determined for a predetermined time period or by a frequency. 

In accordance with the present invention, the recent 
5 feedback information is a weight value learned by the recent 

user feedback or the information of similar images feedback, 
and the whole feedback information is represented by a weight 
value learned by all the feedback given till now. 

Preferably, the multimedia data structure further 
10 comprises recent user feedback reliability information 

representing how reliable the recent user feedback information 
is, and whole feedback reliability information representing how 
reliable the whole feedback information is, wherein the weight 
of the image feature is determined in proportion to 
15 reliabilities of the recent user feedback information, the 

whole feedback information, or both the recent and whole 
feedback information. 

In another aspect of the present invention, there is 
provided a method of determining weights of image features 
20 using multimedia data reflecting a change of a user relevance 

feedback in a system for determining the weights of the 
features used for an image search by the user relevance 
feedback, the method comprising the steps of : 

(a) providing a multimedia data structure including . 
25 information describing the features of a certain image, recent 
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and whole feedback information for the image, and reliability 
inf orma,tion_ of the ' respective feedback information; 

(b) updating the feedback information and their 
reliabilities by learning them in response to the user 

5 feedback ; and 

(c) determining the weights of the image features in 
proportion to reliabilities of the recent feedback information, 
the whole feedback information, or both the recent and the 
whole feedback information. 

10 In accordance with the present invention, the recent 

feedback information is represented by a weight value learned 
by the recent user feedback or the information of feedback 
similar images which is a list form, and the whole feedback 
information is represented by a weight value learned by all the 

15 feedback given till now. 

In accordance with the present invention, the reliability 
of the recent user feedback information is determined in 
proportion to a consistency of a recently used pattern or 
feedback. 

20 In accordance with the present invention, the reliability 

of the whole feedback information is determined in proportion 
to the number of feedback concerned in learning. 

Preferably, the recent user feedback information is 
represented by a similar image list, and a queue algorithm is 

25 used for reflecting the recent user feedback pattern by the 



similar image list. 

In accordance .with the present invention, the queue 
algorithm comprises, the steps of: 

(a) checking whether a similar multimedia object exists in 
5 a current queue when the corresponding object is fed back; 

(b) if it is checked that the corresponding object does 
not exist in the current queue, inputting the corresponding 
object to an uppermost space of a queue entrance, setting the 
number of feedback of the corresponding object to "1", and 

10 shifting objects existing in the queue to lower positions by 

one space; 

(c) if it is checked that the corresponding object exists 
in the current queue, increasing the number of feedback of the 
corresponding object, and shifting the objects existing in the 

15 queue to upper positions by "N" ; and 

(d) if any object is shifted to the lower position over a 
size of the queue at the respective steps, deleting the 
corresponding object from the queue. 

In the queue algorithm according to the present 
20 invention, "N" is fixed, or variably determined in accordance 

with the objects. 

In the queue algorithm according to the present 
invention, "N" is varied in inverse proportion to the frequency 
of appearance of the object. 
25 In the queue algorithm according to the present 



invention, if a similar multimedia object does not exist in the 
■ queue when the similar .'multimedia 'object is '...fed' back, the 
object is inputted to the queue only in case that the object is 
fed back more than the specified threshold number. 

5 

BRIEF DESCRIPTION OF THE DRAWINGS 

The above objects, and other features and advantages of 
the present invention will become more apparent after a reading 
10 of the following detailed description when taken in conjunction 

with the drawings, in which: 

FIG. 1 is a view illustrating the image data structure 
according to the present invention; 

FIG. 2 is a view illustrating the queue structure for 
15 producing a similar image list according to the present 

invention; 

FIG. 3 is a flowchart explaining the similar image list 
managing method using the queue structure of FIG. 2; 

FIG. 4 is a view exemplifying the change of contents of 
20 the queue in the process of FIG. 3; 

FIG. 5 is a view illustrating the list structure for 
producing the similar image list reflecting only the recent 
feedback in the multimedia data structure of FIG. 1; and 

FIG. 6 is a view illustrating the score updating method 
25 applied to the list structure of FIG. 5. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 



Hereinafter, a multimedia data structure reflecting the 
change of the user relevance feedback and a method of 
5 determining weights of image features using this multimedia 

data in accordance with the present invention will be described 
with respect to embodiments illustrated in the annexed 
drawings . 

FIG. 1 is a view illustrating the image data structure 
10 according to the present invention, and shows an example of the 

image data structure to which the technique for learning and 
determining the weights of the image features used for the 
image search according to the user relevance feedback. 

Referring to FIG. 1, an image scheme 101 includes a recent 
15 feedback scheme 102 and a weight scheme 103 . 

The recent user feedback scheme 102 includes a similar 
image lest 104 constituted by the user feedback during the 
recent time (time period or frequency) , and a reliability 105 
representing how reliable the similar image list is. 
20 The similar image list 104 is described by a similar image 

106 and the number of feedback 107 for the image. 

Specifically, the recently used image list is constituted 
by the feedback for the recent similar image, and an item in 
the list comprises the similar image 106 and the number of 
25 feedback for the similar image. 
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Meanwhile, the weight scheme 103 includes a weight 108 
' '. ' learned .by' all the user , feedback -till .now and its reliability 
109. 

Referring to FIG. 1, the reliability for the recent user 
5 feedback can be calculated by the following equation 1. 

[Equation 1] 

, [! (A H 

N 

where, N is the number of feedback, m is the number of 
images in the similar image list, and n± is the number of 
10 feedback given to the i-th image. 

And, the reliability for the weight can be defined by the 
following equation 2 . 

[Equation 2] 

n 

Rel lability — ^ejN k 

k=\ 

15 where, n is the number of authority levels, e k is an 

effective value of the authority level k {e2>e 2 >...>en_ I >e n ) , and N k 
is the number of use by the authority level k. 

In equation 2, n is the number of user authority levels 
(When the feedback is given by judging the similarity of the 

20 image, the reliability for an expert's feedback is much higher 

than that for a non-expert's feedback, and thus the authority 
for a specified user feedback to affect the weight of the 
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feature is graded by level) , e k is an effective value 
representing how- much the • us.er •• authority level affects the 
weight updating value, and N k is the number of feedback by the 
user authority level k (The user authority level becomes 
5 relatively high in case of the expert, and becomes low in case 

of a general user. In case that the user authority level is not 
considered, the reliability is in proportion to the number of 
feedback concerned in learning) . 

The weights of the features can be calculated using the 

10 recent user feedback scheme as described above. When the 

calculated weights and the weights represented in the whole 
information are combined, the weights of the respective 
information are used for the image search in proportion to the 
reliabilities obtained as above. 

15 In the multimedia data structure of FIG. 1, the recent 

feedback scheme 102 is represented as the similar image list. 
However, this can be represented as the feature weight value 
learned by the recent user feedback just like the weight scheme 
103. 

20 Meanwhile, the recent user feedback scheme (or recent use 

pattern) is represented as information by the user pattern or 
feedback given for a predetermined time period, or as 
information by the user pattern or feedback given within a 
recent predetermined frequency. 

2 5 For instance, the recent user feedback scheme is 
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represented as the information fed back for the corresponding 
image within a time- period d between the current time point and' 
a certain time point of the past as the previous images which 
have passed the time period d are deleted, or as the recent 
5 information represented by the images whose number of feedback 

is less than the predetermined threshold number, so that the 
change of the recent user feedback (pattern) can be reflected. 

That is, a consistent user feedback pattern change can be 
considered by eliminating an intermittent or irregular user 

10 feedback and minimizing its influence. 

Specifically, by considering the reliabilities of the 
respective information expressed by the equation 1 or equation 
2 , the weights of the image features are determined in 
proportion to the reliabilities of the recent information, 

15 whole information, or both the recent and the whole 

information, so that the change of the user feedback can be 
sufficiently reflected. 

Also, the weights of the image features are determined in 
a manner that the reliability of the recent information is in 

2 0 proportion to the consistency of the recent use pattern or 

feedback, and the reliability of the whole information is in 
proportion to the number of feedback concerned by learning, so 
that the multimedia object search becomes possible considering 
both the whole feedback information and the recent feedback 

25 information. 
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FIG. 2 is a view illustrating the queue structure for 
. . producing the .similar image lis.t _ -refl.ect.ing only .the ■ .rece.nt ' 
feedback in the multimedia structure • of FIG. 1. 

In the queue structure, an image identification (ID) for 
5 representing a specified image (i.e., similar feedback image) 

and the number of feedback for the corresponding image are 
recorded. Its entrance is on the upper position, and its exit 
is on the lower position. 

FIG. 3 is a flowchart explaining the method of managing 
10 the similar image list reflecting only the recent user feedback 

using the queue structure of FIG. 2 

Referring to FIG. 3, the user feedback image A is received 
at step 301, and it is checked whether the image A exists in 
the queue of FIG. 3. 
15 If it is checked that the input image A is in the queue at 

present, the feedback number of the image A is increased at 
step 303, and then the position of the image A is shifted to 
the entrance of the queue by "N" at step 3 04. 

If it is checked that the input image A is not in the 
2 0 queue, the image A is positioned on the first (uppermost) 

position of the queue. 

The above -described process is repeated, resulting in that 
if the image already existing in the queue is not used for 
feedback quite a while, it is continuously shifted to the lower 
25 position by a new feedback, and then automatically deleted from 
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the queue when it passes over the limited size of the queue. In 
other ..words, a newly inputted image is always- positioned' on "the 
entrance side of the queue, and thus the recent user feedback 
information (similar image list and the number of feedback for 
5 the respective image) can be always provided. 

FIG. 4 shows an example where the contents of the queue is 
changed for the understanding of the process of FIG. 3. In FIG. 
3, N is fixed to 3, and the size of the queue is 10. 

When the images A, B, C, D, and E are sequentially 
10 inputted, the contents of the queue has the form of steps 4 01 

to 405. 

Thereafter, if the image A is fed back at step 406, the 
feedback number of the image A increases to 2, and its position 
is shifted to the queue entrance (uppermost) side by N=3 . 
15 Here, N may be a fixed value, or may be varied according 

to the multimedia object. For instance, it may be varied in 
inverse proportion to the appearance frequency of the object. 

Specifically, if the probability that a certain multimedia 
object is included and appears as the feedback candidate before 
2 0 the user determined the feedback is always 1/2 in comparison to 

other objects, N for this object becomes double in comparison 
to other objects. 

Also, the recent information (i.e., a similar image list 
information) can be managed in a manner that if a similar 
25 multimedia object does not exist in the queue when the object 
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is fed back, the object is unconditionally inputted to the 
first space on the queue entrance,.' the number- of feedback of- ; 
the corresponding object is set as 1, and the current feedback 
object is inputted in the queue only when the object is fed 
5 back over the specified threshold number. 

This can prevent the intermittent feedback from being 
reflected as it is. 

FIG. 5 shows another list structure for producing the 
similar image list reflecting only the recent feedback in the 
10 multimedia data structure of FIG. 1. 

Referring to FIG. 5, an image example 502 included in an 
image example list 501 is composed of its example ID 503, a 
score 504 of the feedback similar image, and a waiting duration 
50 5 representing a time period between the final feedback time 
15 and the present time. 

Here, the waiting duration is determined as A 0' when the 
feedback is received, and increases by '1' whenever every day 
(or every time or every month) has passed or whenever a certain 
feedback arrives . 

2 0 The method of managing the score using such information 

updates the image score in the image list whenever the feedback 
arrives by the following calculation. 

(a) With respect to the image fed back from the similar 

images : 

25 Score (new) = Score (current) X fw (Waiting Duration) + 1 
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(b) With, respect to other images not fed back: 

Score (new) = Score (current) X fw (Waiting Duration)' 
At this time, fW( ) is a function having the 
characteristic that it returns smaller value as the waiting 
duration becomes larger. FIG. 6 shows an example of such a 
function. 

According to the above -de scribed method, the similar image 
not fed back quite a while returns a very small value by fW, 
and it has a small score even if it recorded a large score. 

In the image list, only images over the predetermined 
threshold number on the basis of such a score may be managed by 
list, or only the upper N images may be managed by list. 

As described above, according to the present invention, 
the recent user feedback (pattern) change is reflected 
considering both the recent user pattern (feedback information) 
and the whole feedback information when the image search is 
performed by learning and using the feature weight by the user 
relevance feedback, and thus the image search which accurately 
reflects the user's intention becomes possible. 

Although the preferred embodiments of the present 
invention have been disclosed for illustrative purposes, those 
skilled in the art will appreciate that various modifications, 
additions and substitutions are possible, without departing 
from the scope and spirit of the invention as disclosed in the 
accompanying claims . 



WHAT IS CLAIMED IS: 



l.-A multimedia data structure reflecting change of a user 
relevance feedback for determining weights of image features 
used for an image search, comprising: 

(a) information describing the features of a certain 

image ; 

(b) recent user feedback information when the weights of 
the features of the certain image are learned and determined by 
the user relevance feedback; and 

(c) whole feedback information when the weights of the 
features of the certain image are learned and determined by the 
user relevance feedback; 

(d) wherein the weights of the image features are 
determined considering both the recent user feedback 
information and the whole feedback information, and the image 
features according to the determined weights are used for the 
image search . 

2. The multimedia data structure as claimed in claim 1, 
wherein the recent feedback information and is determined for a 
predetermined time period or by a predetermined frequency. 

3. The multimedia data structure as claimed in claim 1, 
wherein the recent feedback information is a weight value 



learned by the recent user feedback or a similar image 
information, - .and ,the .whole feedback - information' is represented, 
by a weight value learned by all the feedback given till now. 

4. The multimedia data structure as claimed in claim 1, 
further comprising recent user feedback reliability information 
representing how reliable the recent user feedback information 
is, and whole feedback reliability information representing how 
reliable the whole feedback information is. 

5 . The multimedia data structure as claimed in claim 4 , 
wherein the recent user feedback reliability information is 
expressed by 

N 

where, N is the number of feedback, m is the number of 
images in the similar image list, and m is the number of 
feedback given to the i -th image . 

6 . A method of determining weights of image features using 
the multimedia data structure as recited in claim 1, 
comprising : 

representing the recent user feedback information by a 
similar image list; and 



reflecting a recent user feedback pattern by the similar 
. image, list using a" queue algorithm. •' _ ■:; .. 

7. The method as claimed in claim 6, wherein the queue 
5 algorithm comprises the steps of: 

(a) checking whether a similar multimedia object exists in 
a current queue when the corresponding object is fed back; 

(b) if it is checked that the corresponding object does 
not exist in the current queue, inputting the corresponding 

10 object to an uppermost space of a queue entrance, setting the 

number of feedback of the corresponding object to "1", and 
shifting objects . existing in the queue to lower positions by- 
one space ; 

(c) if it is checked that the corresponding object exists 
15 in the current queue, increasing the number of feedback of the 

corresponding object, and shifting the objects existing in the 
queue to upper positions by "N" ,- and 

(d) if any object is shifted to the lower position over a 
size of the queue at the respective steps, deleting the 
2 0 corresponding object from the queue. 

8. The method as claimed in claim 7, wherein "N" is fixed 
or variably determined in accordance with the object. 

25 9. The method as claimed in claim 7, wherein "N" is 
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varied in inverse proportion to the frequency of appearance of 
. . •- " the obj ect . : v, ■ ' 

10. The method as claimed in claim 7, wherein if the 
5 similar multimedia object does not exist in the queue when the 

object is fed back, the object is inputted to the queue only in 
case that the object is fed back more than the specified 
threshold number. 

10 11. The multimedia data structure as claimed in claim 1, 

wherein the recent user feedback information is represented as 
a similar image list, and the similar image list has an image 
list structure composed of a similar image identification, a 
score reflecting the current feedback, and a waiting duration 

15 representing a time period between the final feedback time 

point and the present time point. 

12. A similar image list managing method for the 
multimedia data structure reflecting the user feedback change 
20 as recited in claim 11, comprising: 

managing a similar image list in a manner that only images 
whose scores are over a predetermined threshold number are 
maintained in the list, or only images whose scores are N on 
upper positions of the list if a size of the list is N. 
25 13 . A feedback reflecting score updating method for the 
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multimedia data structure reflecting the recent feedback as 
recited' in claim-' 11, comprising ' updating.- Image scores • in a 
manner that : 

(a) with respect to images fed back from similar images, 
a following calculation is performed; and 

Score (new) = Score (current ) X fw (Waiting Duration) + 1 

(b) with respect to other images not fed back, a 
following calculation is performed; 

Score (new) = Score (current) X fw (Waiting Duration) 

(c) wherein fW (Waiting Duration) is a function having the 
characteristic that it returns smaller value as the waiting 
duration becomes larger. 

14 . A method of determining weights of image features in a 
system for determining the weights of the features used for an 
image search by the user relevance feedback, the method 
comprising the steps of: 

(a) providing a multimedia data structure including 
information describing the features of a certain image, recent 
and whole feedback information for the image, and reliability 
information of the respective feedback information; 

(b) updating the feedback information and their 
reliabilities by learning them in response to the user 
feedback; and 

(c) determining the weights of the image features in 



.proportion to reliabilities of the recent feedback information, 
,.-.■/' the whble ; feedback information, ■' or both the ' recent and •• the 
whole feedback information. 

5 15. The method as claimed in claim 14, wherein the recent 

feedback information is represented by a weight value learned 
by the recent user feedback or a similar image information, and 
the whole feedback information is represented by a weight value 
learned by all the feedback given till now. 

10 

16. The method as claimed in claim 14, wherein the 
reliability of the recent user feedback information is 
determined in proportion to a consistency of a recently used 
pattern or feedback. 

15 

17. The method as claimed in claim 14, wherein the 
reliability of the whole feedback information is determined in 
proportion to the number of feedback concerned in learning. 
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ABSTRACT OF THE DISCLOSURE 

Disclosed is a method of determining weights of image 
features by considering both the recent user relevance feedback 
information and the whole feedback information in a system 
which determines the weights of the image features using the 
user relevance feedback, and a data structure therefor. The 
multimedia data structure reflecting the user feedback change 
includes information describing the features of a certain 
image, recent user feedback information when the weights of the 
features of the certain image are learned and determined by the 
user relevance feedback, and whole feedback information when 
the weights of the features of the certain image are learned 
and determined by the user relevance feedback. The weights of 
the image features are determined considering both the recent 
user feedback information and the whole feedback information, 
and the image features according to the determined weights are 
used for the image search. 
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